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1.  Introduction 

Classification  using  the  MetalMapper  advanced 
electromagnetic  sensor  was  demonstrated  at  the  former 
Camp  Beale,  CA  in  2011.  Camp  Beale  was  also  the  site  of 
the  first  demonstrations  of  handheld  versions  of  advanced 
sensors,  described  in  a  separate  report.  (Ref.  1)  This 
report  summarizes  the  results  of  the  only  the  MetalMapper 
demonstration.  The  document  Implementing  Classification  on 
Munitions  Response  Sites  (Ref.  2)  provides  practical 
information  for  deciding  whether  classification  is 
appropriate  to  a  particular  site  and  how  it  is  best 
implemented. 

Classification  is  motivated  by  the  need  to  perform 
munitions  response  more  cost-effectively  so  that  limited 
clean  up  dollars  can  be  used  to  reduce  real  risk  on 
munitions-contaminated  sites  sooner.  The  estimated 
liability  in  the  FY10  Defense  Environmental  Programs 
Report  to  Congress  for  Munitions  Response  is  $15.2B. 

(Ref.  3)  The  bulk  of  this  liability  is  $10.0B  for  the  1703 
sites  identified  in  the  Formerly  Used  Defense  Sites 
(FUDS)  program  and  $4.4B  for  the  2433  sites  identified 
on  Active  Installations.  The  remaining  $0.8B  is  in  Base 
Realignment  and  Closure  (BRAC).  The  estimated 
completion  dates  for  many  sites,  particularly  in  the  FUDS 
program,  are  decades  out  if  they  are  to  be  cleaned  up  at 
planned  funding  levels  using  current  practice. 

When  a  munitions  response  site  is  cleaned  up,  in  most  cases,  it  is  mapped  with  a  geophysical  sensor 
and  the  locations  of  all  detectable  signals  are  excavated.  Geophysical  sensors  detect  metal  and, 
therefore,  many  of  the  detections  do  not  correspond  to  munitions,  but  rather  to  harmless  metallic 
objects.  Field  experience  indicates  that  95-99%  or  more  of  objects  are  found  to  be  nonhazardous. 
Current  technology  does  not  provide  a  means  to  discriminate  between  munitions  and  other  items, 
termed  “clutter.”  As  a  result,  most  of  the  costs  to  remediate  a  munitions-contaminated  site  using 
current  methods  are  spent  on  excavating  targets  that  pose  no  threat. 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quality  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size  and  aspect  ratio.  The  values  of  these  parameters  may  then  be  used  to 
determine  whether  the  signal  arose  from  a  munition  or  harmless  clutter.  With  reliable  classification, 
only  the  munitions  need  to  be  removed  from  the  site. 


Former  Camp  Beale,  CA  —  complex 
historical  usage,  overlapping  network  of 
ranges  throughout,  relatively  flat  and 
open,  moderate  geologic  interference 

Munitions  —  37 -mm  projectiles,  60 -mm 
mortars,  81  -mm  mortars,  105 -mm 
projectiles 

Kesults  —  MetalMapper  was  used  to 
successfully  classify  all  of  the  targets  of 
interest  and  eliminate  about  75 %  of  the 
clutter.  Production  contractor  field  crews 
from  two  different  vendors  collected  high 
quality  cued  MetalMapper  data.  Both 
production  contractor  geophysicists  and 
the  developers  of  classification  methods 
were  successful  in  using  these  data  to 
achieve  substantial  classification. 

Among  the  production  geophysicists, 
there  was  considerable  variation  in 
performance.  Some  had  difficulty  with 
identifying  about  5-10%  of  the  TOI. 
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The  Strategic  Environmental  Research  and  Development  Program  (SERDP)  and  the  Environmental 
Security  Technology  Certification  Program  (ESTCP)  have  supported  the  development  of  purpose- 
built  advanced  electromagnetic  sensors  and  associated  analysis  methods  for  classification.  Following 
the  successful  demonstration  of  classification  methods  in  controlled  test  environments,  ESTCP 
initiated  a  Classification  Pilot  Program  to  validate  the  application  in  real-world  conditions.  The  goal 
of  the  program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit 
approach,  based  on  principled  analysis  that  is  transparent  and  reproducible.  The  demonstrations  are 
planned  and  conducted  in  cooperation  with  regulators  and  program  managers  in  the  Services. 

The  physics  governing  the  electromagnetic  response  of  a  metal  object  is  well  understood  and 
predictable.  Data  collected  with  these  sensors  contain  the  same  information  content  on  any  site  and 
demonstrations  to  date  have  confirmed  that  classification  works  predictably.  Nevertheless, 
demonstrations  will  be  required  at  a  number  of  sites  to  represent  the  wide  variability  in  munitions 
types,  target  densities,  terrain,  vegetation,  geology,  land  use  history,  future  land  use,  and  other  site 
characteristics  that  will  affect  the  applicability  of  classification  and  to  establish  cost  effectiveness  and 
implementability.  The  demonstrations  also  present  an  opportunity  to  work  out  standard  operating 
procedures  and  establish  quality  control  (QC)  measures.  Prior  demonstrations  have  been  done  at 
the  former  Camp  Sibert,  AL,  the  former  Camp  San  Luis  Obispo,  CA,  and  the  former  Camp  Butner, 
NC.  Details  about  past  and  ongoing  demonstrations  can  be  found  on  the  SERDP-ESTCP  web  site 
at  http:/ / serdp-estcp.org/Featured-Initiatives/Munitions-Response-Initiatives/Classification- 

Applied-to-Munitions-Response. 

The  demonstration  at  the  former  Camp  Beale  is  the  first  test  in  which  production  geophysics 
contractors  both  collected  and  analyzed  advanced  sensor  data  using  the  MetalMapper.  As  such,  one 
purpose  of  the  demonstration  was  to  train  production  contractors  using  the  MetalMapper  for  the 
first  time.  This  is  an  important  consideration  in  evaluating  and  applying  the  results,  which  varied 
widely.  We  discourage  potential  customers  from  using  the  demonstration  results  to  rank  performers 
and  make  contracting  selections.  Particularly,  poor  performance  by  a  contractor  in  a  single 
demonstration  should  not  be  considered  disqualifying  for  performing  classification,  as  analysts  will 
gain  experience  and  improve.  Data  were  also  analyzed  by  experienced  teams  from  the  developers  of 
the  classification  methods.  Table  1  shows  the  participants  and  their  roles  in  the  MetalMapper 
demonstration. 

Table  1.  Participants  in  the  MetalMapper  Demonstration  at  the  former  Camp  Beale 


Task 

Performer(s) 

Task 

Performer(s) 

Site  Preparation 

Parsons 

MetalMapper  Data 
Analysis 

Parsons 

CH2M  HILL 

SAIC 

Sky  Research 

NAEVA  Geophysics 
Dartmouth  College 

Army  Corps  of  Engineers 

EM61-Mk2  Data 
Collection  and  Target 
Selection 

Parsons  (with  input  from 
ESTCP) 

MetalMapper  Data 
Collection 

Parsons/  Geometries 

CH2M  HILL/ Geometries 

Intrusive  Investigation 

Parsons 

Scoring 

Institute  for  Defense 
Analyses 
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2.  Camp  Beale  Demonstration  Flow 

The  sequence  of  the  demonstration  is  outlined  in  the  flow  chart  in  Figure  2-1. 


Figure  2-1.  Flow  chart  outlining  steps  in  the  demonstration  at  Camp  Beale.  Blue  boxes  are  tasks 
performed  by  ESTCP.  Others  are  tasks  performed  by  contractors. 

Prior  to  the  beginning  of  data  collection,  an  instrument 
verification  strip  (IVS)  was  installed  and  the  site  was  seeded 
with  inert  munitions  and  small  industry  standard  objects 
(ISOs),  1-in  nominal  X  4-in  pipe  nipples.  (Ref.  4)  Data 
collectors  visited  the  IVS  twice  daily  to  verify  equipment 
function  at  the  start  and  end  of  each  day.  Since  there  are 
few  native  unexploded  ordnance  (UXO)  on  any  munitions 
response  site,  the  seeds  provided  sufficient  targets  of 
interest  (TOI)  to  allow  a  statistically  defensible 
determination  of  the  correct  classification  of  TOI. 

The  site  was  surveyed  with  an  EM61  to  provide  an  initial 
list  of  detected  anomalies.  The  MetalMapper  was  used  to 
collect  cued  data  over  each  anomaly.  All  detected  targets 


Targets  of  Interest  (TOI)  are  all  objects 
that  must  be  removed  from  the  site. 
Typically  the  TOI  will  include  all 
known  or  suspected  munitions  types ,  any 
other  unexpected  munitions ,  munitions 
parts  such  as  fuyes  that  present  an 
explosive  hazard,  and  all  seeded  items. 
When  classification  is  applied  to  a  site , 
the  local project  team  will  decide  what 
items  constitute  TOI. 
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were  dug  up  to  provide  complete  ground  truth  for  the  purposes  of  determining  performance.  The 
UXO  technicians  photographed  each  item  that  was  dug  and  recorded  its  location,  depth,  and 
description. 

The  geophysical  data  were  passed  to  the  data  analysis  teams.  A  complete  overview  of  the  analysis 
procedures  can  be  found  in  Ref  2.  Briefly,  the  analysts  used  methods  based  on  the  dipole  model  to 
estimate  target  parameters.  Analysts  were  offered  training  data  from  test  pit  measurements  and  the 
opportunity  to  request  additional  training  data  from  the  recovered  targets,  as  though  they  were 
doing  a  limited  number  of  sample  digs.  These  data  were  used  to  set  classifier  rules  —  the  decisions 
that  separate  the  anomalies  into  TOI  and  non-TOI.  The  classifiers  were  then  applied  to  all  of  the 
targets  that  remained  blind  for  each  demonstrator.  Since  training  data  was  by  request,  the  blind 
target  set  was  different  for  each  demonstration. 


The  product  required  from  each  analyst  was  a  ranked  anomaly  list  as  shown  in  Figure  2-2.  One  and 
only  one  judgment  was  required  for  each  entry  on  the  anomaly  list.  The  first  items  on  each  anomaly 
list  are  those  targets  for  which  reliable  parameters  cannot  be  extracted  and  therefore  must  be  dug. 
Next  are  those  items  which  the  analyst  is  the  most  confident  are  TOI.  The  items  are  ranked 
according  to  decreasing  likelihood  that  the  item  is  a  TOI.  Any  items  which  the  analyst  was  able  to 
analyze  but  was  not  able  to  make  a  classification  decision  on  at  this  time  were  placed  next  on  the 
anomaly  list.  Last  are  all  those  items  that  the  analyst  was  confident  are  not  TOI  ranked  by  their 
likelihood.  This  initial  list  is  shown  in  the  left  panel  of  Figure  2-2. 


Initial  Ranked  Anomaly  List 


First 

Pass 

Threshold 


Final  Ranked  Anomaly  List 


Figure  2-2.  Initial  and  Final  Ranked  Anomaly  Lists.  A  detailed  description  is  in  the  text. 


The  seeds  were  divided  into  QC  seeds  and  blind  seeds.  When  analysts  submitted  their  initial 
prioritized  lists,  the  QC  seeds  were  used  to  provide  feedback  if  seed  targets  were  missed.  Analysts 
were  also  provided  with  the  ground  truth  information  on  all  anomalies  in  the  red  part  of  their  lists 
and  any  requested  anomalies  in  the  yellow  part.  This  is  signified  by  the  threshold  on  the  left  side  of 
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Figure  2-2.  Based  on  this  information,  the  analysts  were  then  allowed  to  revisit  their  rankings  and 
assignments  for  all  items  that  were  still  blind  until  they  were  satisfied  that  the  best  possible 
classification  had  been  achieved. 

In  the  final  list,  shown  in  the  right  panel  of  Figure  2-2,  the  analyst  was  required  to  provide  a 
threshold  that  corresponds  to  the  division  between  those  items  recommend  for  digging  and  those 
that  can  safely  remain  in  the  ground.  That  is,  the  list  is  all  red  and  green  with  a  threshold  separating 
the  two  categories.  The  final  prioritized  anomaly  lists  were  scored  against  the  emplaced  blind  seeds 
and  recovered  targets  by  IDA. 

3.  Site  Description  and  Preparation 

The  former  Camp  Beale  is  an  approximately  60,000-acre  site  located  in  Yuba  and  Nevada  Counties, 
CA.  Camp  Beale  was  subject  to  complex  historical  usage  over  many  years  and  there  is  an 
overlapping  network  of  historical  ranges  throughout.  The  demonstration  was  conducted  in  an  area 
that  is  located  within  the  historical  bombing  Target  4  and  the  Proposed  Toss  Bomb  target  area.  An 
aerial  photo  of  the  demonstration  area  is  shown  in  Figure  3-1.  The  area  includes  sections  that  are 
relatively  flat  and  open,  where  the  MetalMapper  demonstrations  took  place,  outlined  in  red  and  blue 
in  Figure  3-1.  The  total  area  for  the  MetalMapper  demonstration  was  6  acres.  Other  sections  with 
trees  and  steeper  slopes  where  MetalMapper  could  not  survey  were  used  in  a  parallel  demonstration 
of  smaller  more  portable  systems.  During  the  demonstration,  it  was  discovered  that  sensor  data 
showed  moderate  geologic  interference. 


o  647,000  647,200  647,400  647,600  o 

o  o 


^  647,000  647,200  647,400  647,600  -f 

UTM  Easting  (m) 


Figure  3-1.  Aerial  photograph  showing  demonstration  area  at  Camp  Beale.  The  MetalMapper  was 
demonstrated  in  the  grid  sections  marked  in  red  and  blue. 

All  visible  metal  objects  were  removed  from  the  surface  at  the  site.  First  order  reference  points  were 
installed  by  a  registered  surveyor  for  geolocation  reference.  A  quiet  area  was  located  near  the 
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demonstration  area  to  establish  an  instrument  verification  strip  (IVS)  used  for  daily  verification  of 
proper  sensor  operation  and  a  training  pit  to  collect  sensor  data  for  algorithm  training. 

Prior  to  the  demonstration,  the  suspected  munitions  in  this  area  included: 


•  37-mm  projectiles, 

•  60-mm  mortars, 

•  81 -mm  mortars,  and 

•  105-mm  projectiles. 


At  the  particular  site  of  this  demonstration,  evidence  of  81 -mm  mortars  and  105-mm  projectiles  was 
found  during  the  Site  Inspection  intrusive  investigation  in  2005.  (Ref.  5)  It  is  also  suspected  that  60- 
mm  mortars  may  be  present.  In  addition,  37-mm  projectiles  have  been  found  scattered  throughout 
the  former  Camp  Beale  and  are  included  as  another  suspected  munition  in  this  area.  Due  to  the 
complex  usage  of  this  site  over  many  years  and  the  overlapping  network  of  historical  ranges,  it  is 
also  possible  that  other  munitions  types  beyond  those  listed  above  are  present. 

The  objective  of  the  demonstration  was  to  detect  and  correctly  classify  all  TOI  on  the  site.  The 
analysts  were  provided  information  about  the  historical  use  and  known  munitions  types.  But,  the 
direction  specified  that,  in  addition  to  these  munitions,  any  unexpected  munitions  would  also  be 
considered  TOI.  During  the  intrusive  investigation,  the  only  munitions-related  items  found,  other 
than  those  expected,  were  expended  fuzes.  These  were  determined  to  be  nonhazardous  by  virtue  of 
their  having  functioned  and  they  were  not  included  in  the  TOI. 

At  a  live  site,  the  number  of  UXO  is  small,  far  from  enough  to  determine  any  demonstrator’s 
classification  performance  with  acceptable  statistical  confidence  bounds.  In  fact,  on  the  Camp  Beale 
demonstration  site,  only  four  munitions  were  recovered  in  the  intrusive  investigation.  Therefore, 
the  site  was  seeded  with  enough  TOI  to  ensure  statistical  validity  on  measures  of  classification  of 
TOI.  The  seeds  are  listed  in  Table  3-1.  For  the  first  time,  the  seeds  included  not  only  inert 
munitions,  but  also  industry  standard  objects.  (Ref.  4)  The  ISOs  are  also  considered  TOI  and 
expected  to  be  both  detected  and  correctly  classified. 


Table  3-1.  Seeds  Emplaced  for  the  Camp  Beale  demonstration  (includes  the 
handheld  demonstration  area) 


Item 

Number 

Depth  range  (cm)* 

Industry  Standard  Object  -  Small 

65 

10-25 

37-mm  projectile 

59 

10-32 

60-mm  mortar 

34 

15-40 

81 -mm  mortar 

33 

15-50 

105-mm  projectile 

9 

15-60 

*Depths  are  to  the  center  of  the  object. 


No  attempt  was  made  to  separate  the  seeds  from  the  surrounding  clutter.  For  safety,  seeds  were 
emplaced  using  standard  anomaly  avoidance  procedures.  For  realism,  the  emplacement  teams  were 
instructed  to  replace  any  metal  dug  up  during  emplacement  back  in  the  hole  with  the  seeded  object. 
All  seed  depths  were  specified  at  1 5  cm  in  the  site  seeding  plan,  with  direction  for  crews  to  bury 
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them  deeper  where  soil  conditions  allowed.  In  some  cases,  the  items  were  buried  at  shallower 
depths  due  to  ground  conditions. 

4.  EM61  Detection  Survey 

An  initial  survey  was  performed  with  an  EM61-MK2  in  its  standard  cart  configuration  with  cm-level 
global-positioning-system  (GPS)  navigation.  These  data  were  used  both  to  provide  a  common 
anomaly  list  for  the  two  MetalMapper  data  collections  that  were  to  follow  and  to  attempt 
classification  using  only  the  EM61  data  as  a  point  of  comparison. 

The  data  quality  objectives  for  the  detection  survey  were  based  on  the  37-mm  projectile,  which  was 
expected  to  be  the  most  difficult  to  detect  TOI  at  the  site.  The  EM61  survey  was  performed  on 
half-meter  line  spacing.  The  anomaly  selection  criteria  were  set  to  detect  a  37-mm  projectile  at  a 
depth  of  one  foot  (30  cm).  This  depth  was  chosen  as  the  deepest  depth  to  which  a  37-mm  could  be 
reliably  detected.  The  EM61  signal  strength  in  channel  2  versus  depth  for  the  37-mm  is  shown  in 
Figure  4-1.  (Ref.  6)  The  signal  at  30-cm  depth  for  the  least  favorable  orientation  is  5.2  mV,  which 
was  used  as  the  amplitude  threshold  for  identifying  anomalies. 


Depth  (cm  bgs)  Assuming  Standard  Wheels 
0  20  40  60  80 


Figure  4-1.  EM61  response  in  Gate  2  versus  depth  for  a  37-mm  projectile 

The  EM61  survey  resulted  in  a  total  of  1490  anomalies  in  the  MetalMapper  area,  including  the  seeds. 
This  translates  to  approximately  250  anomalies  per  acre.  Details  can  be  found  in  the  vendor’s 
report.  (Ref.  7)  All  seeds  were  detected  in  the  EM61  survey. 

5.  MetalMapper  and  Data  Collection 

The  MetalMapper  developed  by  Geometries  is  designed  to  be  a  stand-alone  survey  and  cued 
detection  system.  The  system,  shown  in  Figure  5-1,  is  composed  of  three  orthogonal  1-m  x  1-m 
transmitters  for  target  illumination  and  7  three-axis  receivers  for  recording  the  response.  Its 
sampling  is  electronically  programmable  and  therefore  flexible.  It  measured  the  decay  curve  up  to  8 
ms  after  the  transmitters  were  turned  off.  Centimeter-level  GPS  is  used  for  navigation  and 
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geolocation  and  an  inertial  measurement  unit  (IMU)  is  used  to  measure  platform  orientation.  In 
cued  mode,  MetalMapper  is  positioned  over  each  anomaly  on  its  target  list  and  collects  the  full  suite 
of  data  while  stationary.  The  digital  data  set  produced  by  MetalMapper  is  fully  described  in  Ref.  8. 


Figure  5-1.  Schematic  and  photo  of  the  MetalMapper  as  used  at  former  Camp  Beale 


In  this  demonstration  MetalMapper  was  used  only  in  cued  mode.  It  was  used  in  a  sled  configuration 
mounted  to  a  front  loader  tractor.  Two  commercial  geophysics  vendors,  CH2M  HILL  and  Parsons, 
collected  MetalMapper  data  at  Camp  Beale  for  the  first  time.  Operators  were  trained  in  the  field 
operations  and  QC  procedures  and  data  were  QCed  by  personnel  with  prior  experience  with 
MetalMapper  initially. 

Details  on  the  data  collection  and  QC  procedures  followed  by  each  vendor  can  be  found  in  the 
respective  reports.  (Ref.  9,  10)  The  most  common  QC  failure  was  that  the  MetalMapper  was 
positioned  too  far  from  the  anomaly  to  obtain  reliable  parameter  estimates.  For  both  vendors,  if  the 
separation  between  the  center  of  the  MetalMapper  and  the  anomaly  location  was  more  than  40  cm, 
the  anomaly  was  revisited  and  additional  data  collected  within  the  40-cm  specification. 

The  methods  used  by  the  two  contractors  to  collect  background  data  differed  and  likely  affected  the 
analysis  of  some  targets.  The  background  is  subtracted  from  the  data  taken  over  an  anomaly  to 
remove  any  ground  response.  Parsons  collected  background  data  in  the  morning  and  evening  in  the 
area  of  the  IVS.  CH2M  HILL  collected  background  data  several  times  per  day  at  areas  throughout 
the  site  near  where  they  were  collecting  cued  data.  Due  to  moderate  geologic  response  of  the  soils 
and  variability  in  this  response  across  the  site,  particularly  in  the  central  area  shown  in  Figure  5-2,  the 
more  frequent  background  data  collection  sampling  throughout  the  site  by  CH2M  HILL  resulted  in 
better  background  subtraction  for  some  targets. 

Both  data  sets  were  analyzed  by  multiple  analysts.  Although  there  were  small  differences  in  various 
measures  of  data  quality,  in  addition  to  the  background  subtraction  issue,  both  data  sets  were  of  high 
overall  quality  and  acceptable  to  perform  classification.  Detailed  comparisons  of  the  analyses  of  the 
two  data  sets  can  be  found  in  the  individual  analysts5  reports  available  on  the  SERDP-ESTCP  web 
site  (http:/ / serdp-estcp.org/Featured-Initiatives/Munitions-Response-Initiatives/ Classification- 

Applied-to-Munitions-Response/Former-Camp-Beale). 


Figure  5-2.  Measured  background  variability  in  the  center  four  grids  of  the  site 

The  production  rates  for  the  two  data  collectors  were  115  and  186  targets  per  day  for  CH2M  HILL 
and  Parsons  respectively.  CH2M  HILL  reported  recollecting  data  on  about  7%  of  the  total 
anomalies  due  to  QC  failures,  while  Parsons  reported  an  overall  rate  of  10%,  with  about  20%  at  the 
beginning  of  the  field  work  falling  off  to  only  about  2%  near  the  end.  Both  contractors  noted  that 
these  rates  reflect  inefficiency  inherent  in  learning  to  operate  a  new  system  and  are  not  likely 
predictive  of  future  productivity. 


6.  Classification  Results 

All  data  sets  were  analyzed  by  multiple  analysts,  including  both  the  developers  of  the  analysis 
methods  and  production  geophysics  vendors.  Figure  6-1  shows  an  overview  of  the  results  achieved 
by  all  analysts  working  with  the  MetalMapper  data.  The  panel  on  the  left  shows  the  percent  of  TOI 
correctly  classified  versus  the  number  of  clutter  at  each  analyst’s  operating  threshold.  The  inset 
expands  the  upper  left  of  the  graph  for  clarity.  Desired  performance  is  to  correctly  classify  100%  of 
the  TOI  and  eliminate  all  of  the  clutter.  The  panel  on  the  right  shows  the  number  of  clutter  that  the 
analyst  needed  to  dig  to  get  to  100%  correct  classification  of  the  TOI,  regardless  of  where  the 
analyst  put  the  threshold.  This  can  be  thought  of  as  the  best  the  analyst  could  have  done  by  putting 
the  threshold  in  exactly  the  optimum  place  where  the  last  TOI  is  found  in  the  ranked  list.  The 
symbols  above  the  bars  correspond  to  the  symbols  in  the  left  panel. 

At  their  specified  thresholds,  most  analysts  achieved  good  results,  correctly  classifying  95%  or  more 
of  the  TOI  and  eliminating  70-85%  of  the  clutter.  This  includes  both  the  classification  algorithm 
developers  and  the  production  contractors  and  is  quite  remarkable,  especially  considering  that  this 
was  the  attempt  at  analyzing  MetalMapper  data  by  many  of  these  analysts.  However,  there  was  a 
range  in  performance,  with  one  analyst  correctly  classifying  less  the  95%  of  the  TOI  at  his  threshold 
and  another  achieving  only  about  a  30%  reduction  in  clutter. 
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This  range  is  even  more  evident  in  the  right  panel.  The  best  performer  could  have  eliminated  all  but 
100  of  the  clutter  (a  93%  reduction)  with  100%  correct  classification  of  TOI,  where  as  the  poorest 
performer  at  best  could  have  eliminated  about  only  7%  of  the  clutter. 


Number  of  Clutter  Items  Dug  Analyst 

Figure  6-1.  Results  of  all  analysts  in  the  Camp  Beale  demonstration.  The  left  panel  shows  the 
percent  TOI  correctly  identified  versus  number  of  clutter  at  the  analysts  chosen  threshold.  The  right 
panel  shows  the  number  of  clutter  that  would  have  to  be  dug  to  classify  100%  of  the  TOI,  regardless 

of  where  the  analyst  put  the  threshold. 

6.1  Analysis  by  Algorithm  Developers 

Here  we  present  only  a  few  representative  examples  of  the  results  from  the  developers.  This  is 
meant  to  set  a  baseline  of  what  is  achievable  on  this  site  by  the  most  experienced  analysts.  A 
complete  compendium  of  all  results  can  be  found  in  the  report  by  IDA.  (Ref.  11) 

Figure  6-2  shows  the  receiver  operating  characteristic  curve  achieved  by  SAIC  analyzing  the 
MetalMapper  data  collected  by  Parsons.  The  colors  on  the  plot  correspond  to  the  red  and  green 
colors  in  the  final  ranked  anomaly  list  as  shown  in  Figure  2-2.  The  red  are  the  items  the  analyst 
called  likely  TOI  and  the  green  are  those  the  analyst  called  high  confidence  not  TOI.  No  anomalies 
were  classified  as  “can’t  classify.”  The  graph  plots  the  percent  of  the  targets  of  interest  correctly 
classified  on  the  vertical  axis  and  the  number  of  clutter  items  on  the  horizontal  axis.  The  offset 
from  zero  in  the  starting  point  reflects  any  training  data  that  the  analyst  requested  and  the  initial 
black  line  represents  any  items  in  the  “can’t  analyze”  category.  The  blue  dot  represents  the 
threshold  selected  by  the  analyst  and  the  orange  dot  shows  the  point  on  the  ranked  anomaly  list 
where  100%  of  the  target  of  interest  are  captured.  Ideally,  a  classifier  would  correctly  identify  all 
targets  of  interest  in  the  red  with  zero  clutter  and  all  of  the  clutter  would  be  in  the  green.  In  this 
case,  the  red  part  of  the  curve  would  go  straight  up  to  100%  and  the  green  part  of  the  curve  would 
run  straight  across  the  top  axis.  Success  in  these  demonstrations  was  defined  by  eliminating  the 
maximum  amount  of  clutter  while  correctly  identifying  all  of  the  TOI. 
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Figure  6-2.  Results  of  SAIC  analysis  of  MetalMapper  data  acquired  by 


Parsons 


In  this  demonstration,  there  were  1310  total  clutter  items  as  determined  from  the  ground  truth. 

This  analyst  was  able  to  correctly  identify  all  but  249  of  these  items,  for  a  possible  savings  of  more 
than  80%  of  the  digs.  However,  one  ISO  seed  item  (Target  1965)  is  missed  at  the  analyst  threshold. 
This  target  was  examined  in  detail  to  determine  the  cause  of  the  failure. 

This  analysis  relied  on  matching  the  measured  signature  of  each  target  to  candidate  targets  of 
interest  in  a  library.  Figure  6-3  shows  the  response  of  the  missed  target  (its  three  principle 
polarizabilities  versus  time)  as  calculated  from  the  data  collected  by  Parsons  and  by  CH2M  HILL. 
The  measured  data  are  shown  in  colors  and  the  library  signatures  are  shown  in  gray.  In  the  Parsons 
data,  the  largest  response  in  red  follows  the  library  response  -  although  not  exactly  -  for  the  entire 
time  window,  the  second  response  in  blue  follows  the  library  curve  only  to  about  1  ms  and  then 
wanders  off,  and  the  third  response  appears  poorly  defined  throughout  and  wanders  around.  The 
best  match  of  the  measured  data  was  to  the  ISO,  but  the  fit  was  poor  and  below  the  threshold 
selected  by  the  analyst  to  declare  a  match.  With  closer  visual  QC,  the  poor  data  in  the  green  line 
would  be  discounted,  which  would  have  prevented  the  miss. 

The  responses  calculated  from  the  CH2M  HILL  data  are  shown  for  comparison.  In  this  case,  both 
the  largest  and  second  responses  follow  the  library  curves  throughout  the  time  window.  Again,  the 
weakest  response  is  poorly  defined  and  wanders.  However,  the  overall  match  to  the  library  item  is 
much  better  in  this  case  and  the  target  is  correctly  identified  as  an  ISO.  There  is  nothing  about  this 
ISO  that  makes  it  impossible  to  classify.  However,  the  miss  in  the  analysis  of  one  of  the  data  sets 
highlights  the  difficulties  in  properly  classifying  low  signal  targets. 
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ISO  Match  Metric  =  0.7553  ISO  Match  Metric  =  0.9373 


Figure  6-3.  Polarizabilities  of  Target  1965  derived  from  the  data  collected  by  Parsons(left)  and 
CH2M  HILL  (right).  The  colored  lines  are  the  modeled  polarizabilities  and  the  gray  lines  are  ISO 

polarizabilities  from  the  library 

Figure  6-4  shows  the  results  of  the  Sky  attempt  at  analysis  of  the  EM61  survey  data  only,  which 
support  very  limited  classification.  Only  about  114  of  the  total  1310  total  clutter  items  could  be 
correctly  classified  as  such.  Furthermore,  the  shape  of  the  red  part  of  the  curve  is  far  from  ideal.  It 
rises  steeply  until  only  about  90%  of  the  TOI  are  identified.  Then  it  continues  to  slowly  drift 
upward,  suggesting  that  many  of  the  clutter  and  the  targets  of  interest  look  alike  to  an  EM61.  These 
results  are  typical  of  all  analysts  who  attempted  to  analyze  the  EM61  data  at  Camp  Beale.  This  is  not 
surprising  on  a  site  with  small  targets  of  interest,  ISOs  and  37-mm  projectiles,  that  are  of  similar  size 
to  much  of  the  clutter.  The  information  available  from  the  EM61  is  much  more  limited  than  that 
available  from  the  MetalMapper:  estimates  of  object  size  are  poorer  and  the  decay  rate  is  measured 
at  only  4  coarse  time  intervals  over  a  much  shorter  time  window.  This  performance  is  the  best 
result  that  could  be  achieved  by  an  experienced  analyst  using  these  data. 
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Figure  6-4.  Results  of  Sky  analysis  of  EM61  data 
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6.2  Analysis  by  Production  Geophysics  Companies 

Geophysicists  from  Parsons  and  CH2M  HILL  also  analyzed  the  MetalMapper  data  sets  that  each 
contractor  respectively  collected.  The  results  of  the  Parsons  analysis  are  shown  in  Figure  6-5.  This 
classification  is  excellent.  The  analyst  is  able  to  correctly  identify  all  but  about  285  of  the  total  1310 
clutter  items,  for  a  potential  savings  of  about  75%  of  clutter  digs.  Again,  the  figure  reflects  that 
target  1965,  an  ISO  seed,  was  incorrectly  classified  as  clutter.  This  is  the  same  seed  that  was  missed 
in  the  analysis  by  SAIC  of  the  same  data  set.  The  Parsons  analyst  indicated  similar  difficulties. 
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Figure  6-5.  Results  of  Parsons  analysis  of  MetalMapper  data  acquired  by  Parsons 


Figure  6-6  shows  the  results  of  the  CH2M  HILL  analysis  of  their  MetalMapper  data.  These  results 
show  the  right  general  trend,  in  that  the  red  part  of  the  curve  rises  steeply  initially  and  the  green  part 
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flattens  out.  But,  this  analyst  placed  a  large  number  of  anomalies  in  the  yellow  “can’t  classify” 
category  and  it  is  clear  he  had  difficulty  with  a  number  of  TOI  that  appear  in  the  green  “high 
confidence  clutter”  category.  We  do  not  believe  that  this  is  due  to  any  major  differences  in  the 
quality  of  the  MetalMapper  data  collected,  as  other  analysts  in  the  developer  community  achieved 
good  results  with  the  CH2M  HILL  data.  Rather,  we  believe  that  additional  training  is  needed  on  the 
analysis  side  so  that  all  analysts  can  spot  quality  problems  in  the  numerous  steps  that  are  involved  in 
the  classification  process. 
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Figure  6-6.  Results  of  CH2M  HILL  analysis  of  MetalMapper  data  acquired  by  CH2M  HILL 


7.  Cost  Comparison 

The  demonstration  took  place  on  a  small  part  of  the  former  Camp  Beale  site  and  incurred  costs  for 
many  items  specific  to  a  demonstration  that  would  not  be  needed  in  an  application  of  classification 
to  a  real  site.  Nevertheless,  we  can  extract  meaningful  projected  performance  for  the  technology 
and  apply  reasonable  industry  unit  costs  for  various  elements  to  arrive  at  a  total  cost  comparison  for 
clearing  an  example  100-acre  site  with  and  without  the  use  of  classification. 

We  made  the  following  assumptions: 

•  The  example  takes  place  in  an  area  with  similar  munitions  types  and  the  same  density  of 
anomalies  as  seen  in  the  demonstration.  Excluding  the  seeds,  there  were  1338  anomalies  in 
the  6-acre  demonstration  area.  Extrapolating,  we  would  expect  about  22300  anomalies  in  a 
similar  100-acre  area,  with  22266  clutter. 

•  Two  TOI  were  found  during  the  intrusive  investigation  of  the  6-acre  area,  for  a  rate  of  0.33 
TOI  per  acre.  In  a  10-acre  site  we  predict  about  33  native  TOI. 

•  The  baseline  is  an  EM61  survey  with  0.5-m  line  spacing.  This  would  be  used  to  select 
anomalies  for  digging  without  classification  and  the  same  anomalies  would  be  interrogated 
with  MetalMapper  and  classified. 


1  This  number  differs  slightly  from  the  1310  anomalies  that  were  scored  above  because  there  are  28  instances  where 
there  were  two  detection  anomalies  that  required  investigation,  but  only  one  object  was  recovered. 
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•  The  site  is  seeded  at  a  rate  so  on  average  one  seed  will  be  encountered  each  day  of 
MetalMapper  data  collection.  With  an  estimate  of  22,300  total  anomalies  and  a  production 
rate  of  200  anomalies  per  day,  we  seed  a  conservative  125  inert  items.  These  QC  seeds 
would  be  used  whether  classification  was  used  on  the  site  or  not. 

•  The  classification  performance  is  as  achieved  by  Parson,  as  shown  in  Figure  6-5,  with  —75% 
of  the  clutter  correctly  identified  and  remaining  undug. 

•  The  unit  costs  are  as  shown  in  Table  8-1. 


Table  8-1.  Unit  cost  assumptions 


Item 

Units 

Cost 

Digs 

Per  hole 

$125 

MetalMapper  Classification 

Per  anomaly 

$25  and  $35 

EM61  Survey  Data  Collection  and  Analysis 

100  acres 

$94,000 

Seed  Emplacement 

125  seeds 

$22650 

With  these  assumptions  the  costs  were  calculated  using  the  elements  shown  in  Table  8-2.  If 
classification  can  be  done  from  $35  per  anomaly  including  both  data  collection  and  analysis,  which  is 
consistent  with  the  projections  of  the  production  companies  based  on  this  demonstration,  a  45% 
overall  savings  is  possible.  If  this  cost  can  be  lowered  to  $25  to  classify  each  anomaly,  the  potential 
project  savings  increases  to  52%. 


Table  8-2.  Cost  Comparison  for  100  acres  of  comparable  Camp  Beale  site 


Item 

j  No  Classification  | 

j  Classification 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

($35  per  MM) 

Cost/$ 

($25  per  MM) 

Seeds 

125  items 

65,600 

125  items 

65,600 

65,600 

EM61  Survey 

100  acres 

94,000 

100  acres 

94,000 

94,000 

MetalMapper 

Classification 

n/a 

0 

22425 

anomalies 

784,875 

560,625 

Seeds  Dug 

125 

15,625 

125 

15,625 

15,625 

Native  UXO  Dug 

33 

4,181 

67 

4,181 

4,181 

Clutter  Dug 

22,266 

2,783,250 

5677 

695,830 

695,830 

TOTAL 

2,920,075 

1,617,461 

1,393,211 

Percent  Savings 

45% 

52% 

8.  Conclusions 

Classification  was  used  on  the  former  Camp  Beale  to  successfully  identify  all  of  the  TOI  and 
eliminate  about  75%  of  the  clutter.  Production  contractor  field  crews  from  two  different  vendors 
collected  high  quality  cued  MetalMapper  data.  Both  production  contractor  geophysicists  and  the 
developers  of  classification  methods  were  successful  in  using  these  data  to  achieve  substantial 
classification.  Among  the  production  geophysicists,  there  was  considerable  variation  in 
performance.  Some  had  difficulty  with  identifying  about  5-10%  of  the  TOI.  Additional  training  is 
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needed  so  that  all  analysts  are  able  to  identify  problems  that  can  be  encountered  in  the  multiple 
analysis  steps  required  in  classification. 

In  two  ways  the  former  Camp  Beale  is  a  challenging  site.  First,  the  TOI  included  small  37-mm 
projectiles,  as  well  as  seeded  small  ISO’s  for  the  first  time.  These  items  are  similar  in  size  to  the 
much  of  the  clutter.  Second,  the  geology  at  the  site  produced  moderate  soil  responses  that  affected 
the  data  analysis.  Anomaly  density  was  moderate,  at  about  250  per  acre,  and  did  not  present  any 
particular  difficulty. 

Attempts  at  classification  using  the  EM61-MK2  were  not  successful  at  this  site,  likely  due  to  the 
requirement  to  classify  small  objects  of  a  size  comparable  to  much  of  the  clutter.  This  is  consistent 
with  the  results  of  the  prior  demonstration  at  the  former  Camp  Butner,  where  both  the  presence  of 
small  targets  and  high  anomaly  density  led  to  poor  classification  using  the  EM61  data. 

9.  Acronyms 

BRAC 
ESTCP 
FUDS 
GPS 
IMU 
ISO 
IVS 
QC 

SERDP 
TOI 
UXO 
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